feat(enterpriseops): add external MCP benchmark integration - #4944

What does this PR do?

Adds our standalone EnterpriseOps-Gym integration to current main, extracted from 0f4aa2c738b1694bf1c020db5a7ef0bfe204e440 on dmolinari/officegen and based on 9334fa27c53d2e35e1242bb9661fad605dc8cc84.

The integration runs the pinned 649-task public oracle split across eight domains through Gym agents, a resources server that seeds per-rollout databases and proxies MCP tools, and the upstream SQL/judge verifier semantics. It includes benchmark preparation, trusted endpoint/replica configuration, cleanup retries, tool-schema snapshots, an optional per-turn telemetry agent, and runtime/provenance documentation.

This is the external-service variant: operators run the seven pinned MCP containers and configure seed_sql_root and trusted endpoints. Related #2142 develops managed sandbox startup. The focused cleanup contribution in #4774 remains targeted at that PR's branch. This separate main-based draft makes our complete implementation available for review; no separate issue is needed for this extraction.

Main-branch compatibility additions include the task-data schema, explicit environment-server declarations, explicit non-Ray runtime declarations, and the janitor/reseed race guard from #4774.

Validation

• gym eval prepare --benchmark enterpriseops: 649 rows prepared from dataset revision c8e538eae8a6205294f0a86675fefdc1fac408f6; exact counts: calendar 61, CSM 103, drive 64, email 67, HR 102, ITSM 103, teams 61, hybrid 88. All 649 rows pass task-data validation with no unknown, conflicting, or misplaced fields.
• gym env test +entrypoint=resources_servers/enterpriseops_gym +should_validate_data=true +skip_venv_if_present=true: 52 passed, example-data validation passed.
• gym env test +entrypoint=responses_api_agents/turn_logging_agent +should_validate_data=true +skip_venv_if_present=true: 4 passed.
• Focused core tests for benchmark discovery, resources-server registration, agent registration, and task-data schemas: 512 passed. The repository-wide catalog-listing test initially failed because this Mac's hostname cannot be resolved for four unrelated apex_shortlist/hle sandbox configurations; it passes with a test-process-only host-resolution override to a documentation address. No DNS or repository workaround was committed.
• Benchmark and telemetry configs resolve with error_on_agent_without_environment_server=true, using a placeholder model config and no inference.
• pre-commit run --all-files and git diff --check: passed. The private guardwords hook reported its catalog unavailable and skipped its scan.
• Environments were installed from current main's lockfile with uv sync --locked --extra dev; each server has its own environment. DCO sign-off is included. Full repository CI has not been run locally.

Rollouts

Fresh real-model smokes on 2026-10-22 used azure/openai/gpt-5-mini, temperature 0, a 30-step cap, and an 8,192-token per-response cap against the existing real CSM Docker service through this integration's configured endpoint and SQL seed root.

Both runs used public example task_20251205_153330_906_a8eea1c0_8c7a6205:

The inspected trajectories update the entitlement, create the critical case with the expected assignment, and link its resolution SLA. SQL verification confirms the resulting state. Both runs also injected a first deletion failure: the database handle was retained, then the real janitor deletion succeeded and removed the session.

These are two rollout executions of one task, not a full-benchmark evaluation. Existing parity/performance/RL-pilot reports retain their historical provenance and were not rerun for this draft.

Compatibility and benchmark impact

No existing environment is replaced. The new configs retain verified: false. Reward defaults to upstream name-collapsed parity scoring; strict every-verifier metrics are emitted separately and can be selected for reward. Task preparation remains pinned and writes its output atomically. The large JSON portion of the diff consists primarily of seven public tool-schema snapshots; the full 649-task benchmark is downloaded at prepare time and is not committed.

Draft review follow-ups include the optional telemetry agent's divergence from current SimpleAgent cookie merging/no-tool termination, whether to use native trajectory capture instead, and runtime enforcement of selected/restricted tools beyond conversion-time filtering. The external service deployment choice also needs review alongside #2142.

Checklist

• Focused EnterpriseOps extraction; unrelated OfficeGen and AutomationBench changes excluded.
• Component tests and example-data checks pass; core-test environment limitation documented above.
• Runtime and benchmark documentation included; historical evidence distinguished from fresh smokes.
• Pre-commit passes, with the unavailable private guardwords catalog noted above.
• New Python sources include Apache-2.0 SPDX headers.
• DCO sign-off included.